add `Era_Run3_2024`, `Era_Run3_2025`, Tier0 reco scenario for 2024, and change default era in `PyReleaseValidation` #43761

mmusich · 2024-01-20T14:02:38Z

PR description:

In response to #43756, following #41271 (comment):

add Era_Run3_2024, Era_Run3_2025 (currently straight copies of Run3)
add Tier0 reco scenario for 2024
change default era in PyReleaseValidation for 2024 MC workflows

PR validation:

runTheMatrix.py --what upgrade -l 12824.0 runs fine
scram b runtests runs fine:

Creating test log file logs/el9_amd64_gcc12/testing.log
Pass    2s ... Configuration/DataProcessing/TestConfigDP_1
Pass   14s ... Configuration/DataProcessing/TestConfigDP_10
Pass   12s ... Configuration/DataProcessing/TestConfigDP_11
Pass  150s ... Configuration/DataProcessing/TestConfigDP_12
Pass  131s ... Configuration/DataProcessing/TestConfigDP_2
Pass   24s ... Configuration/DataProcessing/TestConfigDP_3
Pass   33s ... Configuration/DataProcessing/TestConfigDP_4
Pass  273s ... Configuration/DataProcessing/TestConfigDP_5
Pass   98s ... Configuration/DataProcessing/TestConfigDP_6
Pass  155s ... Configuration/DataProcessing/TestConfigDP_7
Pass  228s ... Configuration/DataProcessing/TestConfigDP_8
Pass   11s ... Configuration/DataProcessing/TestConfigDP_9
Pass    1s ... Configuration/PyReleaseValidation/test-das-selected-lumis
Pass  175s ... Configuration/PyReleaseValidation/test-runTheMatrix-interactive

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

N/A

cmsbuild · 2024-01-20T14:03:01Z

cms-bot internal usage

cmsbuild · 2024-01-20T14:07:09Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-43761/38481

This PR adds an extra 44KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): Add run3 tracking low pu era #33532, Porting Pixel Tracks to Alpaka [Not to Merge] #41117, Guard for No Digis for Phase2 and Patatrack NuGun #41255, ECAL unpacker and ECAL multifit algorithm migration to alpaka #43257, Pixel Alpaka Migration: Configs and Fixes [VII] #43294, Introduce matrix workflows for Alpaka-based Particle Flow #43701, Phase2-hgx352 Modify the scenarios D104, D105 to accommodate modified T35 and I17 (also go back to C17 for D105). Add a new scenario D106 with C23 (same as C22 but without cell definition in Geant4) #43760
- File Configuration/StandardSequences/python/Eras.py modified in PR(s): Add run3 tracking low pu era #33532

cmsbuild · 2024-01-20T14:07:34Z

A new Pull Request was created by @mmusich (Marco Musich) for master.

It involves the following packages:

Configuration/DataProcessing (operations)
Configuration/Eras (operations)
Configuration/PyReleaseValidation (upgrade, pdmv)
Configuration/StandardSequences (operations)

@davidlange6, @miquork, @sunilUIET, @srimanob, @AdrianoDee, @antoniovilela, @fabiocos, @rappoccio, @cmsbuild, @subirsarkar can you please review it and eventually sign? Thanks.
@sameasy, @makortel, @GiacomoSguazzoni, @missirol, @VinInn, @felicepantaleo, @dgulhan, @VourMa, @rovere, @JanFSchulte, @Martin-Grunewald, @fabiocos, @mmusich, @AnnikaStein, @mtosi, @slomeo this is something you requested to watch as well.
@rappoccio, @antoniovilela, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

mmusich · 2024-01-20T15:18:25Z

Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py

@@ -2534,7 +2534,7 @@ def condition(self, fragment, stepList, key, hasHarvest):
 class UpgradeWorkflow_DDDDB(UpgradeWorkflow):
    def setup_(self, step, stepName, stepDict, k, properties):
        the_era = stepDict[step][k]['--era']
-        if 'Run3' in the_era  and '2023' not in the_era and 'Fast' not in the_era and "Pb" not in the_era:
+        if 'Run3' in the_era and '2023' not in the_era and '2024' not in the_era and 'Fast' not in the_era and "Pb" not in the_era:


I am not totally sure to understand what's the intent here, but this line is necessary to avoid having runtime matrix failures. I am wondering if instead of excluding explicitly every "customization" on top of Run3 one should just match exactly Run3 and be done with it (but then I don't understand the purpose of this class).
Perhaps @cms-sw/pdmv-l2 or @cms-sw/upgrade-l2 can clarify.

@srimanob do you have any idea of the usage of this UpgradeWorkflow_DDDDB. Sincerely I don't know/remember.

This is to use DDD from DB, see
https://github.com/cms-sw/cmssw/blob/master/Configuration/AlCa/python/autoCondModifiers.py#L34-L75

It was invented when we move DDD to DD4hep, but we wanted to keep DDD around for 2022. That is why we have not in in several places to avoid failure due to non-existing era. I agree with @mmusich, we can just match exactly.

mmusich · 2024-01-21T08:28:41Z

@cmsbuild, please test

cmsbuild · 2024-01-21T12:27:51Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ddf07d/36952/summary.html
COMMIT: dffa7fb
CMSSW: CMSSW_14_0_X_2024-01-20-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/43761/36952/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 2 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3247526
DQMHistoTests: Total failures: 3
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3247501
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 200 log files, 161 edm output root files, 48 DQM output files
TriggerResults: no differences found

malbouis · 2024-01-22T11:21:45Z

Configuration/DataProcessing/python/RecoTLR.py

@@ -95,6 +95,11 @@ def customisePostEra_Run3_2023(process):
    customisePostEra_Run3(process)
    return process

+def customisePostEra_Run3_2024(process):
+    #start with a repeat of 2023
+    customisePostEra_Run3_2023(process)


Maybe here could be a repeat of Run3 instead, with customizePostEra_Run3? This way it would not call the run3_egamma_2023 modifier. Would it make sense?

This way it would not call the run3_egamma_2023 modifier.

This is not the way to not call it. These are just copies of copies from Run-2.
If you want to avoid calling run3_egamma_2023 it needs to be done in the era definition itself (though it defies somewhat the purpose of the PR).

done in the last push, since there is still interest in something similar for #44970

cmsbuild · 2024-05-22T11:23:06Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ddf07d/39464/summary.html
COMMIT: 880ab43
CMSSW: CMSSW_14_1_X_2024-05-21-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/43761/39464/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially removed 187 lines from the logs
Reco comparison results: 2 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3338862
DQMHistoTests: Total failures: 3
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3338839
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 202 log files, 165 edm output root files, 48 DQM output files
TriggerResults: no differences found

vlimant · 2024-05-31T07:16:43Z

if there are no objections, I think we should move this forward ASAP do that #44970 can be added for data taking

mmusich · 2024-05-31T07:20:21Z

@cms-sw/pdmv-l2 @cms-sw/upgrade-l2 can you please have a look and comment if this update is acceptable on your side?

srimanob · 2024-05-31T07:46:12Z

+Upgrade

AdrianoDee · 2024-05-31T08:12:18Z

+pdmv

mmusich · 2024-06-05T18:56:44Z

@cms-sw/orp-l2 kind ping on this PR

vlimant · 2024-06-10T07:05:26Z

ping @cms-sw/orp-l2

rappoccio · 2024-06-12T01:35:57Z

+1

cmsbuild · 2024-06-12T01:36:17Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will be automatically merged.

patinkaew · 2024-06-24T13:14:35Z

Hi @mmusich, do we have a backport of this to CMSSW14_0_X?

mmusich · 2024-06-24T13:18:29Z

@patinkaew

do we have a backport of this to CMSSW14_0_X?

here it is #45291, thanks for the reminder (I had actually prepared the branch some time ago).

cmsbuild added this to the CMSSW_14_0_X milestone Jan 20, 2024

cmsbuild added operations-pending pending-signatures tests-pending orp-pending pdmv-pending upgrade-pending code-checks-pending labels Jan 20, 2024

cmsbuild added code-checks-approved and removed code-checks-pending labels Jan 20, 2024

mmusich commented Jan 20, 2024

View reviewed changes

mmusich mentioned this pull request Jan 20, 2024

Question on Era of Run-3 2024 workflows #43756

Closed

cmsbuild added tests-started and removed tests-pending labels Jan 21, 2024

cmsbuild added tests-approved and removed tests-started labels Jan 21, 2024

malbouis reviewed Jan 22, 2024

View reviewed changes

This was referenced Jan 23, 2024

ECAL unpacker and ECAL multifit algorithm migration to alpaka #43257

Merged

Split testConfigDP into smaller parts #43776

Merged

Pixel Alpaka Migration: Configs and Fixes [VII] #43294

Merged

mmusich force-pushed the mm_dev_add_Era2024_2025 branch from dffa7fb to e48530c Compare January 26, 2024 09:07

cmsbuild added tests-pending code-checks-pending and removed tests-approved code-checks-approved labels Jan 26, 2024

cmsbuild added tests-approved and removed tests-started labels May 22, 2024

cmsbuild mentioned this pull request May 22, 2024

Alpaka implementation of Hcal Local Reconstruction (Mahi) #44910

Merged

AdrianoDee mentioned this pull request May 27, 2024

2024 Data RelVals and InputInfo Events Skimming (actually) Working #45055

Merged

cmsbuild added upgrade-approved and removed upgrade-pending labels May 31, 2024

cmsbuild added pdmv-approved and removed pdmv-pending labels May 31, 2024

cmsbuild mentioned this pull request Jun 3, 2024

CMSSW Integration of LST #45117

Merged

3 tasks

cmsbuild added operations-approved fully-signed orp-approved and removed operations-pending pending-signatures orp-pending labels Jun 12, 2024

cmsbuild merged commit 00b743f into cms-sw:master Jun 12, 2024
11 checks passed

mmusich deleted the mm_dev_add_Era2024_2025 branch June 12, 2024 06:33

mmusich mentioned this pull request Jun 17, 2024

Unprescaled AXO seeds (L1_AXO_Nominal, L1_AXO_VTight) added to the un… #45242

Merged

mmusich mentioned this pull request Jun 24, 2024

[14.0.X] add Era_Run3_2024, Era_Run3_2025, Tier0 reco scenario for 2024, and change default era in PyReleaseValidation #45291

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add `Era_Run3_2024`, `Era_Run3_2025`, Tier0 reco scenario for 2024, and change default era in `PyReleaseValidation` #43761

add `Era_Run3_2024`, `Era_Run3_2025`, Tier0 reco scenario for 2024, and change default era in `PyReleaseValidation` #43761

mmusich commented Jan 20, 2024 •

edited

Loading

cmsbuild commented Jan 20, 2024 •

edited

Loading

cmsbuild commented Jan 20, 2024

cmsbuild commented Jan 20, 2024

mmusich Jan 20, 2024

AdrianoDee Jan 31, 2024

srimanob Feb 1, 2024 •

edited

Loading

mmusich commented Jan 21, 2024

cmsbuild commented Jan 21, 2024

malbouis Jan 22, 2024

mmusich Jan 22, 2024

mmusich May 22, 2024

cmsbuild commented May 22, 2024

vlimant commented May 31, 2024

mmusich commented May 31, 2024

srimanob commented May 31, 2024

AdrianoDee commented May 31, 2024

mmusich commented Jun 5, 2024

vlimant commented Jun 10, 2024

rappoccio commented Jun 12, 2024

cmsbuild commented Jun 12, 2024

patinkaew commented Jun 24, 2024

mmusich commented Jun 24, 2024 •

edited

Loading

add Era_Run3_2024, Era_Run3_2025, Tier0 reco scenario for 2024, and change default era in PyReleaseValidation #43761

add Era_Run3_2024, Era_Run3_2025, Tier0 reco scenario for 2024, and change default era in PyReleaseValidation #43761

Conversation

mmusich commented Jan 20, 2024 • edited Loading

PR description:

PR validation:

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

cmsbuild commented Jan 20, 2024 • edited Loading

cmsbuild commented Jan 20, 2024

cmsbuild commented Jan 20, 2024

mmusich Jan 20, 2024

Choose a reason for hiding this comment

AdrianoDee Jan 31, 2024

Choose a reason for hiding this comment

srimanob Feb 1, 2024 • edited Loading

Choose a reason for hiding this comment

mmusich commented Jan 21, 2024

cmsbuild commented Jan 21, 2024

Comparison Summary

malbouis Jan 22, 2024

Choose a reason for hiding this comment

mmusich Jan 22, 2024

Choose a reason for hiding this comment

mmusich May 22, 2024

Choose a reason for hiding this comment

cmsbuild commented May 22, 2024

Comparison Summary

vlimant commented May 31, 2024

mmusich commented May 31, 2024

srimanob commented May 31, 2024

AdrianoDee commented May 31, 2024

mmusich commented Jun 5, 2024

vlimant commented Jun 10, 2024

rappoccio commented Jun 12, 2024

cmsbuild commented Jun 12, 2024

patinkaew commented Jun 24, 2024

mmusich commented Jun 24, 2024 • edited Loading

add `Era_Run3_2024`, `Era_Run3_2025`, Tier0 reco scenario for 2024, and change default era in `PyReleaseValidation` #43761

add `Era_Run3_2024`, `Era_Run3_2025`, Tier0 reco scenario for 2024, and change default era in `PyReleaseValidation` #43761

mmusich commented Jan 20, 2024 •

edited

Loading

cmsbuild commented Jan 20, 2024 •

edited

Loading

srimanob Feb 1, 2024 •

edited

Loading

mmusich commented Jun 24, 2024 •

edited

Loading